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Objects are reminiscent of actions often performed with them: knife and apple remind us 
on peeling the apple or cutting it. Mnemonic representations of object-related actions 
(action codes) evoked by the sight of an object may constrain and hence facilitate 
recognition of unrolling actions. The present fMRI study investigated if and how action 
codes influence brain activation during action observation. The average number of action 
codes (NAC) of 51 sets of objects was rated by a group of n = 24 participants. In an fMRI 
study, different volunteers were asked to recognize actions performed with the same 
objects presented in short videos. To disentangle areas reflecting the storage of action 
codes from those exploiting them, we showed object-compatible and object-incompatible 
(pantomime) actions. Areas storing action codes were considered to positively co-vary 
with NAC in both object-compatible and object-incompatible action; due to its role in 
tool-related tasks, we here hypothesized left anterior inferior parietal cortex (aIPL). In 
contrast, areas exploiting action codes were expected to show this correlation only in 
object-compatible but not incompatible action, as only object-compatible actions match 
one of the active action codes. For this interaction, we hypothesized ventrolateral premotor 
cortex (PMv) to join aIPL due to its role in biasing competition in IPL. We found left anterior 
intraparietal sulcus (IPS) and left posterior middle temporal gyrus (pMTG) to co-vary with 
NAC. In addition to these areas, action codes increased activity in object-compatible action 
in bilateral PMv, right IPS, and lateral occipital cortex (LO). Findings suggest that during 
action observation, the brain derives possible actions from perceived objects, and uses 
this information to shape action recognition. In particular, the number of expectable actions 
quantifies the activity level at PMv, IPL, and pMTG, but only PMv reflects their biased 
competition while observed action unfolds. 
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INTRODUCTION 

Observed action entails a highly complex stimulus that prompts 
a multitude of attentional and memory processes. The observer 
has to be flexible with regard to potential actions that may unroll, 
but yet quickly discard those which do not pertain to the actual 
situation. How is this achieved? 

When considering object-related action, the observer has 
access to at least two sources of information that usually help 
him to quickly recognize the most probable action goal: manip- 
ulation movements and objects. These two basic sources of 
information, rather than being complementary, are intimately 
interrelated: familiar objects such as mobile phones or knifes 
are strongly reminiscent of manipulations that we perform 
with them everyday. Hence, the observer's brain may use these 
automatically evoked memories of distinct object-related actions 



(action codes, hereafter) to bias or constrain expectation on 
upcoming manipulations and hence facilitate recognition of the 
action, i.e., implemented object function, and thereby the prob- 
able actor's goal. For instance, when seeing someone handling 
a knife and an apple, the object set "knife, apple" evokes two 
action codes: "cutting apple with knife" and "peeling apple 
with knife." While tracking the unfolding manipulation, we at 
a point in time notice that the peeling-action code matches the 
observed manipulation, and recognize the actor is peeling the 
apple with the knife (object function), probably to prepare it for 
eating (goal). 

The present fMRI study focused on automatically evoked 
object-related action codes to find out if, and if so how, they influ- 
ence the neural basis of action observation. In order to recognize 
an observed action, it would make no sense to match the observed 
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action to all possible action memories we have. Rather, one could 
suggest that objects automatically evoke mnemonic codes of the 
handful of actions we most frequently perform with them (i.e., 
action codes) (Helbig et al., 2006; Myung et al., 2006; Campanella 
and Shallice, 2011), information that could greatly constrain the 
number of expectable actions. Note that we very quickly recog- 
nize objects (Bar, 2003), including their pragmatic properties (Liu 
et al, 2009; Proverbio et al., 2011), while observed manipulation 
only unfolds and disambiguates in time. 

Although the notion of action codes is reminiscent of what 
is called object affordance, they differ in an important respect. 
According to the classical concept of affordance (Gibson, 1977; 
see McGrenere and Ho, 2000 for modifications and alternatives), 
an object affords actions in that seeing the object can automat- 
ically prime, and hence facilitate, object-compatible actions in 
a particular observer (depending also on the observer's body); 
the object does so by virtue of its physical properties: e.g., size 
and shape of the object afford appropriate grasping, and its loca- 
tion appropriate pointing (Tucker and Ellis, 1998, 2001, 2004; 
Craighero et al, 1999; Pavese and Buxbaum, 2002; Phillips and 
Ward, 2002; Derbyshire et al, 2006; Symes et al, 2007; Cho and 
Proctor, 2010; Pellicano et al, 2010; Iani et al, 2011; McBride 
et al., 2012). In contrast, we here were interested in object- 
evoked representations of actions that do not derive from the 
object's size, shape or orientation, but from associative memo- 
ries of how and what for we use these objects in everyday life. 
Note that our manipulation did not dissociate this "how" and 
"what for," which can be doubly dissociated in patient groups 
(Buxbaum and Saffran, 2002). 

To show that action codes are effective during action obser- 
vation, we should find that it makes a difference how many 
action codes are currently evoked, and whether the observed 
action matches one of them or not. Accordingly we should 
find (HI) increased activity in areas that code for currently 
active action codes; and (H2) increased activity in areas that 
exploit them for action recognition. In order to disentangle 
these effects, we presented object-compatible (normal) action 
and object-incompatible (pantomime) action. As an example for 
an incompatible action, the actor performed the movements for 
"cracking an egg" while holding and moving an orange and 
an orange squeezer. Object-compatible and object-incompatible 
actions were performed on objects whose NACs, i.e., number of 
possible action codes related to them, were assessed in a pre-fMRI 
rating study (see Methods and Figure 1). 

The NAC effect (HI) should be only driven by the perceived 
object(s) but be independent of the actually observed action. 
Thus, we considered areas that positively co-vary with the NAC 
during object-compatible and object-incompatible action to clas- 
sify as areas storing action codes. In contrast, currently evoked 
action codes can only be exploited for action recognition (H2) 
when observing the former, but not the latter. That means, only 
if the observed actor executes one of the currently evoked action 
codes, i.e., in object-compatible actions, can the observer benefit 
from their automatic pre-activation. 

Thus, these action codes put an effective constraint on the to- 
be-expected possible actions, and identification of the matching 
action will be enhanced. In terms of neural computation, this 



results in a continuous, top-down reinforcement of the match- 
ing action code, in competition to all currently active action 
codes, during ongoing action observation (see neuroanatomi- 
cal hypotheses below). This reinforcement may be achieved by 
enhancement of the matching action code, or by inhibition of 
the currently competing but non-matching action codes, or both. 
Since the present approach could not distinguish between these 
options, we will refer to this mechanism shortly as "reinforce- 
ment" hereafter. 

It is particularly essential that, in order to interpret an area's 
activation as exerting a reinforcement of one particular action 
code among all currently evoked and hence competing action 
codes, rather than simply signaling for a successful matching, 
this activation has to depend on competition strength: to make 
the particular matching action code to come out on top of three 
possible actions (NAC 3) is more demanding than on top of 
two actions or only one (NAC 2 or 1, respectively). Accordingly, 
regarding (H2), we were not interested in the main effect of object 
compatibility, but rather in the interaction between the NAC and 
object compatibility: We considered areas that positively co-vary 
with the NAC during object-compatible significantly more than 
during object-incompatible action to classify as areas exploiting 
the currently active action codes. These areas should show a sig- 
nificant parametric effect of NAC in object-compatible actions, 
no significant parametric effect of NAC in object-incompatible 
actions, and a significant interaction between the NAC and object 
compatibility. 

Regarding the neural correlates of action code storage (HI), 
we hypothesized that activity in the left anterior inferior pari- 
etal lobule (aIPL) increases with the NAC, no matter whether 
the movie shows an object-compatible or an action-incompatible 
manipulation. During tool-related tasks, left aIPL is often seen 
in co-activation with left ventral premotor cortex (PMv) and the 
posterior middle temporal gyrus (pMTG) (Johnson-Frey, 2004; 
Culham and Valyear, 2006; Martin, 2007; Creem-Regehr, 2009), 
i.e., exactly the same network that is reported for action obser- 
vation (Grezes and Decety, 2001; Van Overwalle and Baetens, 
2009; Caspers et al., 2010), but also for action execution, action 
imagery, action planning, and action imitation. This network 
has been referred to as MNS (mirror neuron system) or AON 
(action observation network), but due to the spectrum of action- 
related roles of this triad, the label "Action Network" might 
be more generic. Regarding our hypothesis on areas housing 
action codes (HI) we focused on aIPL because of converging 
findings from various studies reporting left aIPL to be engaged 
in the representation of pragmatic properties of objects, partic- 
ularly manipulation knowledge (e.g., Chao and Martin, 2000; 
Kellenbach et al, 2003; Johnson-Frey, 2004; Rumiati et al, 2004; 
Boronat et al., 2005; Ishibashi et al., 2011). In her thorough 
review, Creem-Regehr (2009) proposed to conceive of the inferior 
IPL/IPS as a region for motor cognition, including the genera- 
tion of internal representations for action and knowledge about 
actions. Patient studies indicate that the ability to retrieve the cor- 
rect manipulation for a given tool can be selectively impaired, 
while in the same patient, the ability to correctly name the tool 
or point to the tool when named by the experimenter are pre- 
served (Ochipa et al., 1989). This defect in tool utilization has 



Frontiers in Psychology | Perception Science 



June 2014 | Volume 5 | Article 636 | 2 



Schubotz et al. 



Object-suggested actions in action observation 



Pre-experimental rating data: 
Mean Number of Action Codes (NAC) related to object/object sets 




Object / object set No. 1 to 51 



FIGURE 1 | Rating values for the number of action codes (NAC) related to 
51 objects/object sets as assessed pre-experimentally in an independent 
group of n = 24 volunteers. Values were subsequently used to model the 
BOLD fMRI amplitude during object-compatible and object-incompatible 
action. Some values are highlighted and correspond to the objects shown in 



the photos, based on which participants delivered their ratings. For instance, 
the object set "pencil and sharpener" were mostly reported to be related to 
"sharpening a pencil" (resulting in NAC=1), whereas "candle and lighter" 
participants often associated "lighting a candle," and sometimes also "melting 
a candle's foot" (in order to firmly fixate it in a candle stand) (NAC=1.65). 



been coined limb apraxia (Rothi and Heilman, 1997). In spite 
of considerable variance between findings, evidence converges 
that patients with impaired object use and pantomiming to visu- 
ally presented objects mostly suffer from lesions that include the 
left IPL (Rumiati et al, 2004). This region is considered cru- 
cial for gestural praxis, tool knowledge, body part knowledge, 
and manipulation knowledge, together coined as the ability to 
generate internal models of object-interaction (Buxbaum et al., 
2005). 

The frontal component of the Action Network, the left PMv, 
was expected to respond quite differently than alPL. In relation 
to our second question, whether there would be areas reflecting 
the selection among competing action codes, we hypothesized 
(H2) the left PMv to be enhanced by the number of action codes, 
but in contrast to alPL only for object-compatible, not object- 
incompatible action videos. That is, we should see left PMv only 
for the interaction between NAC and object compatibility of the 
observed action. 

This hypothesis was motivated by the notion that premotor 
regions serve the top-down selection among alternative manipu- 
lation options provided in parietal areas (Fagg and Arbib, 1998; 
Rushworth et al., 2003). The lateral premotor cortex is made of 
a variety of functionally highly specialized sub-areas which in 
turn are connected in multiple parallel, largely segregated loops 
to a mosaic of sub-areas making of the parietal cortex (Luppino 
and Rizzolatti, 2000). Among these premotor-parietal loops, the 
ventral premotor — inferior parietal loop was reported to code 
for grasping and manipulation (Rizzolatti et al, 1987), but also 
for the sight of graspable objects (via so-called canonical neu- 
rons in PMv, Murata et al., 1997; Rizzolatti and Fadiga, 1998). 
As for fronto-parietal loops in general, the functional role of PMv 
with regard to IPL is providing inhibitory and reinforcing input 



to focus and elevate currently relevant codes in IPL to modulate 
adaptive perception, attention and behavior. 

Addressing the interplay between lateral premotor and pari- 
etal areas in object-directed action, Fagg and Arbib (1998) put 
forward that anterior parietal cortex provides ventral premo- 
tor cortex with a multiple description of how the object can 
be grasped and used. In PMv, then, all corresponding motor 
acts are first activated, and then the currently required one is 
selected (or reinforced, to keep with the more process-dynamic 
notion adopted above). For instance, neurophysiological stud- 
ies in macaques indicate that potential plans for movements to 
multiple targets are simultaneously represented in parietal and 
frontal areas (Andersen and Cui, 2009) and, as information accu- 
mulates, eliminated in a competition for overt execution (Cisek 
and Kalaska, 2005) (for an application of the notion of selec- 
tion as frontoparietal reinforcement signal in humans, see e.g., 
Ramsey et al., 2013). Fagg and Arbib (1998) proposed that in 
action execution, this selection needs prefrontal input (via pre- 
supplementary motor area) that signals the current goals of 
the individual. However recent imaging findings indicate that 
action selection that emerges from the race between competi- 
tive decision-units is reflected in premotor, not prefrontal, areas 
(Rowe et al., 2010), suggesting that action selection in premotor 
sites does not necessarily need prefrontal bias. 

In the present experimental approach, competition between 
the action codes evoked by the perceived object was to be resolved 
by the actually observed manipulation. We expected that in case of 
a successful match (which was only possible for object-compatible 
actions), the PMv would reinforce the matching action codes 
in alPL, just as it does during action execution. Load on this 
reinforcement would be a function of action codes only in 
object-compatible action, as outlined above, manifesting in an 
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Examples for objects, their NAC and presented action 
(snapshots of videos) 



1 NAC 



2 NAC 





objects: pencil, sharpener 
1 NAC: sharpening pencil 
video: cutting fruit 



objects: knife, orange 

2 NACs: peeling/cutting orange 

video: sharpening pencil 




objects: toothbrushApaste 
1 NAC: applying toothpaste 
video: applying toothpaste 



objects: knife, apple 

2 NACs: peeling/cutting apple 

video: cutting an apple 



FIGURE 2 | Snapshots of videos showing actions with objects mostly 
related to either one (left panel) or two (right panel) actions. Actions 
could be exploited to constrain action recognition only in object-compatible 
actions (lower panel; examples show "applying toothpaste" on the left and 
"cutting an apple" on the right), but not in object-incompatible actions 
(upper panel; examples show "cutting a fruit" on the left and "sharpening a 
pencil" on the right). The corresponding videos can be found in the 
supplementary material. 



interaction of the NAC and object compatibility of the observed 
action. Of course, if PMv does exert an action code dependent 
reinforcing signal on aIPL during action observation, this effect 
should be reflected in both of these areas. 

Finally, in object-incompatible action, reinforcement load 
should be generally higher than in object-compatible action, as 
action recognition is unrestricted by the currently evoked action 
codes: there are objects that evoke action codes, but none of them 
matches the observed action. That does also mean, reinforce- 
ment load should not depend on the number of action codes 
in case of object-incompatible actions. Since objects employed 
in object-incompatible actions evoke action codes that are not 
effective to constrain the matching process, the number of pos- 
sible actions is the number of all possible actions that humans 
do perform with objects. Accordingly, our third hypothesis (H3) 
was that object-incompatible actions lead to an overall higher 
response than object-compatible actions in left PMv and IPL 
(replicating Schubotz and von Cramon, 2009), but show no pos- 
itive co-variance with the number of currently activated action 
codes. 

MATERIALS AND METHODS 
PARTICIPANTS 

Seventeen right-handed, healthy volunteers (13 women, 20-31 
years, mean age 25.6 years) participated in the study. After being 
informed about potential risks and screened by a physician of 
the institution, participants gave informed consent before partic- 
ipating. The local ethics committee of the University of Cologne 
approved the experimental standards. Data were handled anony- 
mously. 

STIMULI AND TASKS 

Subjects were presented with two kinds of trials, videos showing 
actions (snapshots in Figure 2; for examples of videos, see supple- 
mentary material) and short verbal action descriptions (without 
video) referring to these actions (e.g., "cutting bread," "peeling an 
apple," "cleaning a cell phone"). Each trial lasted 6 s and started 
with a movie (2 s) followed by a fixation phase (Figure 3). The 
length of the fixation phase (2.5-4 s) depended on the variable jit- 
ter times (0, 500, 1000, or 1500 ms) that were inserted before the 
movie to enhance the temporal resolution of the BOLD response. 
Actions were either performed on appropriate objects (object- 
compatible actions, e.g., peeling an apple with a knife) or on 
inappropriate objects (object-incompatible actions, e.g., making 
the same movements with a pencil and a sharpener). 

Subjects were instructed to attend to the presented movies. 
They were informed that some of the movies were followed by 
a trial providing a verbal action description that either matched 
or did not match the content of the preceding movie. Subjects 
then performed a verification task, i.e., they were asked to indi- 
cate by button press whether the verbal description was consistent 
with the action movie previously presented or not. It was empha- 
sized that it did not play any role whether actions, to which 
the action description referred, were object-compatible or not. 
Thus, when subjects saw the action "peeling an apple" performed 
with an apple and a knife, or with a pencil and a sharpener, 
and the subsequent trial delivered the verbal action description 



"peeling an apple," the correct answer was "yes." In the case of an 
action description trial, participants immediately delivered their 
responses on a two-button response box using their index finger 
for affirmative responses (description pertains to the movie in the 
preceding trial) and their middle finger for rejections (descrip- 
tion did not pertain to the movie in the preceding trial). Half 
of the action descriptions were to be affirmed and half to be 
rejected. 

Action movies varied with regard to the number of action 
codes (Figure 1; see also "Pre-experimental assessment on 
objects' average number of action codes"). Importantly, in case 
that two or three objects were involved in an action, they always 
made up object sets that were indicative of possible actions 
(which were, of course, not actually performed in the case of 
object-incompatible actions); e.g., participants were presented 
the object-incompatible action "cracking an egg" performed in 
an as-if manner on the objects "orange" and "orange squeezer," 
i.e., a pair of objects that could be used to prepare orange juice 
(Figure 2). Thus, videos showing object-incompatible actions 
never involved meaningless object sets such as e.g., an orange and 
a sharpener. 

Twenty percent of the movies (i.e., 21 of 105 object-compatible 
actions and 21 of 105 object-incompatible actions) were fol- 
lowed by an action description that had the length of a regular 
trial (2 s description, including response phase, plus 4 s fixation 
phase), resulting in 42 additional trials. Finally, 20 empty trials 
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video (2 s) fixation (2.5 - 4 s) 




open bottle? + 

question (2 s) fixation/response (2.5 -4 s) 

FIGURE 3 | Example sequence of five trials with within-trial time 
course for one video trial (top) and for one question trial (bottom). 



(resting state) of 6 s duration were presented intermixed with the 
experimental trials. Thus, 272 trials were presented altogether. 

For each subject, each action was presented four times dur- 
ing the course of the experimental session, two times object- 
compatible and two times object-incompatible, with different 
objects each time. Importantly, we balanced the order of appear- 
ance of object-compatible and incompatible actions in the time 
course of the appearance. Hence, all combinations (1: compatible, 
compatible, incompatible, incompatible; 2: compatible, incom- 
patible, compatible, incompatible; 3: incompatible, compatible, 
incompatible, compatible; 4: incompatible, incompatible, com- 
patible, compatible) occurred equally often in the experiment. 

PRE-EXPERIMENTAL ASSESSMENT ON AVERAGE NUMBER OF ACTION 
CODES 

In order to determine the NACs of the objects later used in 
the action movies, we assessed the spontaneous assignment of 
actions to these objects in a group of n = 24 volunteers. To 
avoid mnemonic confounds, this group was not identical to 
the group tested in the fMRI session, i.e., none of the partic- 
ipants of the pre-experimental assessment was included in the 
fMRI study. Participants were given photographs of 51 objects 
(e.g., cell phone) or object sets (e.g., apple and knife). There 
were 27% single objects, 63% two-object sets and 10% 3-object 
sets. Participants were asked to write down all potential actions 
that the presented objects were typically reminiscent of in their 
eyes. For instance, participants rated an apple and a knife to 
be most suggestive of "cutting an apple into halves," "peeling 
an apple," and "coring an intact apple" (3 actions), whereas 
an orange and an orange squeezer were rated suggestive of 
"squeezing an orange" (1 action). To assess NACs rather than 
object affordances, participants were explicitly asked to pro- 
vide object-specific goal-directed actions, not object grasping or 
transport. 



We did not impose a temporal restriction onto this rating pro- 
cess and participants had time to thoroughly ponder on potential 
actions. The collection of actions typically took less than 1 min 
per object or object set; moreover, no participant came up with 
invalid or odd actions. On the basis of this rating, the aver- 
age NAC score was calculated for each object or set of objects 
(Figure 1). NAC scores, ranging from 0.95 to 2.73, were subse- 
quently used in the parametric analysis of fMRI data (see below). 
Importantly, there was no systematic relation between NAC score 
and number of objects in a set. Thus, single objects yielded a mean 
NAC of 1.63 ± 0.33, two-object-sets 1.66 ± 0.4, and three-object 
sets 1.18 ± 0.1. To statistically rule out the potential confound that 
NAC co-vary with the number of objects displayed in an action, 
we calculated a correlation of NAC with the number of objects. 
There was no correlation [rug) = — 0.223, p = 0.12]. 

MRI DATA ACQUISITION 

Twenty-two axial slices (192 mm field of view; 64 x 64 pixel 
matrix; 4 mm thickness; 1 mm spacing; in-plane resolution of 
3x3 mm) parallel to bi-commissural line (AC-PC) covering 
the whole brain were acquired using a single-shot gradient EPI 
sequence (2 s repetition time; 30 ms echo time; 90° flip angle; 
116 kHz acquisition bandwidth) sensitive to BOLD contrast. 
Prior to the functional imaging, 26 anatomical Tl-weighted 
MDEFT images (Ugurbil et al, 1993; Norris, 2000) with the 
same spatial orientation as the functional data were acquired. 
In a separate session, high-resolution whole-brain images (160 
slices of 1 mm thickness) were acquired from each partici- 
pant to improve the localization of activation foci using a Tl- 
weighted 3-D-segmented MDEFT sequence covering the whole 
brain. 

fMRI DATA ANALYSIS 

After offline motion-correction using the Siemens motion pro- 
tocol PACE (Siemens, Erlangen, Germany), fMRI data were pro- 
cessed using the software package LIPSIA (Lohmann et al, 2001). 
To correct for the temporal offset between the slices acquired 
in one image, a cubic-spline interpolation was employed. Low- 
frequency signal changes and baseline drifts were removed using 
a temporal high-pass filter with a cutoff frequency of 1/90 Hz. 
Spatial smoothing was performed with a Gaussian filter of 
5.65 mm FWHM (SD = 0.8 voxel). To align the functional data 
slices with a 3-D stereotactic coordinate reference system, a rigid 
linear registration with six degrees of freedom (three rotational, 
three translational) was performed. 

The rotational and translational parameters were acquired 
on the basis of the MDEFT slices to achieve an optimal match 
between these slices and the individual 3-D reference dataset. The 
MDEFT volume dataset with 160 slices and 1-mm slice thickness 
was standardized to the Talairach stereotactic space (Talairach 
and Tournoux, 1988). The rotational and translational parame- 
ters were subsequently transformed by linear scaling to the same 
standard size. The resulting parameters were then used to trans- 
form the functional slices employing a trilinear interpolation, so 
that the resulting functional slices were aligned with the stereo- 
tactic coordinate system. Resulting data had a spatial resolution 
of 3 x 3 x 3 mm (27 mm3). 
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The statistical evaluation was based on a least-squares estima- 
tion using the general linear model for serially auto-correlated 
observations (Friston et al, 1995; Worsley and Friston, 1995). 
The design matrix was generated with a delta function, convolved 
with the hemodynamic response function (gamma function) 
(Glover, 1999). The design matrix comprised the following 
events: object-compatible action videos, object-incompatible 
action videos, object-compatible action videos with an amplitude 
modeled by the corresponding objects' NAC, object-incompatible 
action videos with an amplitude modeled by the correspond- 
ing objects' NAC, question trials, and empty trials (null 
events). 

Brain activations were analyzed time-locked to onset of the 
videos. The model equation, including the observation data, the 
design matrix, and the error term, was convolved with a Gaussian 
kernel of dispersion of 4 s FWHM to account for the temporal 
autocorrelation (Worsley and Friston, 1995). In the following, 
contrast images, that is, beta value estimates of the raw-score 
differences between specified conditions were generated for each 
participant. As all individual functional datasets were aligned to 
the same stereotactic reference space, the single-subject contrast 
images were entered into a second-level random effects analysis 
for each of the contrasts. 

One-sample t-tests were employed for the group analyses 
across the contrast images of all participants that indicated 
whether observed differences between conditions were signifi- 
cantly distinct from zero. The f- values were subsequently trans- 
formed into z-scores. To correct for false-positive results, an 
initial z-threshold was set to 2.33 (p < 0.01, one-tailed). In a sec- 
ond step, the results were corrected for multiple comparisons at 
the cluster level, using cluster size and cluster value thresholds 
that were obtained by Monte-Carlo simulations at a significance 
level of p = 0.05, i.e., the reported activations were significantly 
activated atp < 0.05, corrected for multiple comparison at clus- 
ter level. 

RESULTS 

BEHAVIORAL RESULTS 

Performance was assessed by error rates and reaction times. 
We calculated paired-f-tests (one-tailed, in expectation of 
lower performance in object-incompatible actions) for each 
of these measures between question trials addressing object- 
incompatible and object-compatible actions. While error rates 
became not significant (t\(, = —0.66, p = 0.259), reaction times 
showed a small effect (fig = 1.75, p = 0.049). Thus, object- 
compatible and object-incompatible actions were responded to 
equally correct (mean ± SE: object-compatible action 5.8 ± 
1.2% errors and object-incompatible 4.8 ± 1.4% errors), but 
recognition of object-incompatible actions took 40 ms longer 
(object-compatible action 1192 ± 61 ms and object-incompatible 
1232 ± 69 ms). 

Moreover, we calculated bivariate correlations between NACs 
and reaction times or error rates, respectively. As a result, there 
was neither an effect on error rates [r(\e) = —0.012, p = 0.96], 
nor on reaction times [f(i6) = —0.026, p = 0.92]. Together, 
behavioral statistics suggested that recognition times were 
slightly but significantly reduced by object information, but both 



object-incompatible and object-compatible actions could be 
successfully identified. 

As a caveat, we employed a retrospective judgment in order to 
control for the participants' performance in action recognition. 
This task was implemented by extra question trials that followed 
an action observation trial in order to avoid response-related 
confounds: motor execution, and, even worse, trial-specific inter- 
actions between executed button press, implied and observed 
manipulations. That is, reaction times and error rates refer 
to a response delivered one trial after action observation. So, 
our paradigm was optimized for fMRI rather than for specific 
behavioral effects. 

fMRI RESULTS 

Our two main hypotheses HI and H2 addressed the paramet- 
ric effect of number of action codes. In left aIPL, this effect 
was expected to be independent of the object-compatibility of 
the observed action (Hypothesis HI), but depend on object- 
compatibility in PMv (Hypothesis H2). HI was tested by calcu- 
lating a conjunction of the thresholded parametric contrast in 
object-compatible actions and the thresholded parametric con- 
trast in object-incompatible actions. H2 was tested by an interac- 
tion contrast, i.e., by contrasting the parametric effect of number 
of action codes in object-compatible actions with the parametric 
effect of number of action codes in object-incompatible actions. 
While H2 addressed compatible > incompatible actions, we also 
report the reverse contrast for exploratory reasons. We follow 
the view that in order to show that differences between the two 
parametric effects (number or action codes in object-compatible 
actions, number of actions in object-incompatible actions) are 
statistically significant, it is not enough to show each of them, 
but to calculate a contrast between both, i.e., an interaction 
(Nieuwenhuis et al, 2011). In order to make their respective con- 
tributions to these two effects descriptively transparent, however, 
we also report the parametric effect in object-compatible actions 
and the parametric effect in object-incompatible actions sepa- 
rately. Finally, we report findings on the main effect of object 
compatibility of observed action (Hypothesis H3). 

Parametric effect of number of action codes (NAC) common to 
object-compatible and object-incompatible actions (Hypothesis HI) 

The conjunction between the thresholded parametric effect of 
action codes in object-compatible actions and the thresholded 
parametric effect of action codes in object-incompatible actions 
revealed activation in the left alPS and in the left posterior middle 
temporal gyrus (pMTG). Both areas showed increasing activation 
with the number of possible actions associated with the objects 
shown in the movies (Figure 4A, Table 1). 

Interaction effect of NAC and object-compatibility (Hypothesis H2) 

Parametric effect of NAC in object-incompatible actions. For 

object-incompatible action, activity increased with number of 
action codes in the left alPS, the left pMTG, encroaching into 
adjacent pSTS and TPJ, and in the left Cuneus (Figure 4D, 
Table 2). 

Parametric effect of NAC in object-compatible actions. For 

object-compatible action, activity increased with number of 
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Parametric effects of number of object-related action codes (NAC) 



A NAC in object-compatible n object-incompatible action 
(conjunction) 




C NAC in object-compatible action D NAC in object-incompatible action 

PMv alPL/plPS pMTG/LO alPL/plPS PMv pSTS/TPJ alPL/plPS pMTG 

^1 




PMv 



NAC in object-compatible > object-incompatible action 
(interaction) 



FIGURE 4 | The mere sight of objects triggered the representation of a 
number of possible object-related actions that quantified the activation 
in several cortical areas [P < 0.05, corrected for multiple comparisons). 

(A) Left aIPL and pMTG can be considered to code for the number of 
object-related action codes (NAC), as their activity increased with the NAC 
regardless of whether the observed action was object-compatible (and hence 
matched one of the evoked NACs) or object-incompatible (cf. Hypothesis H1). 

(B) In contrast, PMv, right IPS, left pIPS, bilateral LO and mid-insula increased 
with NAC only in object-compatible action, presumably reflecting a top-down 



competition bias between the observed and the remaining object-related but 
unobserved actions (cf. Hypothesis H2). Interestingly, left pSTS/TPJ 
increased with NAC in object-incompatible actions blue spot in (B) suggesting 
that in case of a non-match, lower constraints on expectable actions (i.e., 
higher NACs) increased efforts to read out the actor's hand postures and 
movements. For descriptive purposes, (C) and (D) show the NAC effect 
separately for videos on object-compatible and object-incompatible 
(pantomime-with-incompatible-objects) actions. Table 1 lists Talairach 
coordinates for z-maps shown in (A) and (B), Table 2 for (C) and (D). 



action codes in the Action Network (PM, aIPL, pMTG) as well 
as the fusiform gyrus/lateral occipital cortex and mid-insula. All 
activation spots were found in both hemispheres (Figure 4C, 
Table 2). 

Interaction effect of NAC in object-compatible > incompatible 
actions. For object-compatible action, additional activation was 
found to increase with the NAC in right ventral and dorsal pre- 
motor cortex (PMv, PMd) as well as in bilateral mid-insula. 
Moreover, activation in IPS was recorded bilaterally and extended 
from anterior into its horizontal segments. Finally, activation 
in the pMTG extended inferiorly and posteriorly into the lat- 
eral occipital cortex (LO) and emerged particularly in the right 
hemisphere (Figure 4B, Table 1). 

Interaction effect of NAC in object-incompatible > object- 
compatible actions. Some areas responded to increasing number 



of evoked action codes exclusively during object-incompatible 
actions. These were located in the left pSTS, extending posteri- 
orly and dorsally into the temporo-parietal junction (TPJ), and 
in left cuneus (Figure 4B, Table 1). 

Main effects of observation of object-compatible and 
object-incompatible action (Hypothesis H3) 

The present study employed object-compatible and object- 
incompatible action to investigate the effects of action codes and 
their impact on action observation. However, it is important to 
consider that all effects so far reported supervened on the typ- 
ical network found for action observation (cf. Introduction), 
including the lateral premotor-parietal loops as well as temporo- 
occipital areas related to attention to motion, movements and 
objects (Figure 5A). Recorded activation patterns were almost 
identical for object-compatible and object-incompatible actions 
when compared to rest, but direct contrasts revealed significant 
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Table 1 | Anatomical area (for abbreviations, see main text), Talairach 
coordinates (x, y, z) and maximal Z-score (max) of activated clusters 
(p = 0.05, corrected for multiple comparisons) for parametric effects 
of the number of automatically evoked object-related action codes 
(NAC) that were common to both for object-compatible and object- 
incompatible actions (conjunction; Hypothesis H1) or interacted 
with object compatibility of observed action (Hypothesis H2) 
(cf. Figures 4A,B). 

Area x y z Max 

HI: NAC IN BOTH OBJECT COMPATIBLE n INCOMPATIBLE ACTION 

alPS -62 -24 36 3.08 

pMTG -47 -66 9 2.83 

H2: INTERACTION NAC x OBJECT COMPATIBLE > INCOMPATIBLE 
ACTIONS 



PMv 


-56 


9 


27 


3.08 




52 


9 


24 


4.43 


alPS 


-50 


-27 


36 


4.18 




40 


-27 


42 


5.22 


pIPS 


-35 


-54 


63 


3.66 




31 


-45 


66 


4.61 


Mid-lnsula 


-38 


-6 


21 


3.24 




34 


-3 


34 


3.42 


LO / pMTG 


37 


-69 


-3 


4.07 


Cuneus 


22 


-90 


6 


5.10 


pSTS / TPJ 


-59 


-57 


15 


-3.65 


Cuneus 


-8 


-81 


0 


-4.81 



Table 2 | Anatomical area (for abbreviations, see main text), Talairach 
coordinates (x, y, z) and maximal Z-score [max) of activated clusters 
(p = 0.05, corrected for multiple comparisons) for parametric effects 
of the number of automatically evoked object-related action codes 
(NAC), separately analyzed for object-compatible actions and for 
object-incompatible actions (cf. Figures 4C,D). 

Area x y z Max 

NAC IN OBJECT-COMPATIBLE ACTIONS 



alPS/SMG 


-47 


-24 


33 


5.38 




49 


-21 


33 


5.67 


pMTG, LO 


-47 


-63 


3 


3.51 




43 


-63 


0 


4.79 


Fusiform gyrus / LO 


-47 


-75 


-6 


3.06 


Fusiform gyrus 


46 


-45 


-12 


3.18 


pi PS 


-32 


-51 


60 


5.74 




28 


-48 


60 


5.39 




37 


-30 


39 


5.91 


PMv 


-56 


6 


27 


4.23 




52 


9 


27 


5.29 


PMd 


22 


-12 


51 


3.64 


Mid-lnsula 


-38 


3 


12 


4.55 




31 


-6 


18 


4.77 


NAC IN OBJECT-INCOMPATIBLE ACTIONS 


alPS 


-62 


-24 


33 


3.15 


pMTG/pSTS/TPJ 


-53 


-69 


21 


4.13 


Cuneus 


-14 


-78 


-3 


4.59 



differences in the modulations of this network as well, particularly 
enhanced activity in left PMv and IPL for object-incompatible 
actions (Figure 5B). For object-compatible actions, we found 
enhanced activity solely in fusiform areas (Figure 5C). These 
findings fully replicate those of a previous study (Schubotz and 
von Cramon, 2009; Figure 3). 

In contrast to the findings in Schubotz and von Cramon 
(2009), object-incompatible actions in the current study addi- 
tionally activated mesial Brodmann Area (BA) 8, the ventral 
tegmental area and the bilateral dorsal anterior insula (not shown 
in Figure 5). These activations probably reflect dopaminergic 
enhancement during decision uncertainty (Volz et al, 2005). 
We consider this difference to be due to the use of object sets 
that always implied valid action options for object-compatible 
as well as for object-incompatible actions, in contrast to our 
previous study. Thus, uncertainty was somewhat higher for 
object-incompatible, as objects did not indicate whether the to- 
be-expected action would be an object-incompatible or an object- 
compatible action, and action codes could not become effective to 
constrain the process of identifying the observed action. 

DISCUSSION 

Objects are reminiscent of actions that we typically perform with 
them. These object-related actions (action codes) may influence 
action observation by providing a constraint on the number of 
expectable actions, and hence facilitate action recognition. We 
used fMPJ in an action observation paradigm to test whether 
left aIPL codes for action codes, i.e., whether its activation level 



varies as a function of the currently evoked number of action 
codes (main effect action codes; Hypothesis HI). Moreover, 
we employed object-compatible and object-incompatible action 
videos to test whether left PMv reflects the exploitation of evoked 
action codes. Here we reasoned that an area that exploits action 
codes in action observation should positively co-vary with the 
NAC in case of object-compatible, but not object-incompatible 
action, since action codes can act as a constraint only in the 
former (interaction effect action codes x object compatibility; 
Hypothesis H2). 

In expectation to replicate findings from a previous study 
(Schubotz and von Cramon, 2009), we hypothesized that object- 
compatible and object-incompatible action differ in highly sim- 
ilar way from the resting level, but when directly contrasted 
with one another show enhanced activity for object- incompatible 
actions in the entire Action Network, including left PMv and IPL 
(Hypothesis H3). 

RESPONSES TO AUTOMATICALLY EVOKED CODES OF 
OBJECT-RELATED ACTIONS 

Object-compatible and object-incompatible actions differed with 
respect to the usability of object information, but objects implied 
possible actions in both. To tap this object-based action-pre- 
activation, we computed the parametric effect of the number of 
action codes separately for object-compatible actions and object- 
incompatible actions, and subsequently built the conjunction of 
both. As a result, activity was recorded in only two areas, the left 
aIPL and the left pMTG (Figure 4A). Finding aIPL confirmed our 
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compatible n incompatible 
action vs. rest 



incompatible > compatible 
action 




compatible > incompatible 
action 




FIGURE 5 | Significant activation differences addressed by Hypothesis 
(H3) (P < 0.05, corrected for multiple comparisons) that replicate 
previous findings (Schubotz and von Cramon, 2009). (A) Main effects of 
observation of object-compatible and object-incompatible action as 
compared to rest (conjunction). (B) and (C): Differential effects as revealed 
by direct contrasts between observation of object-compatible and 
object-incompatible action. Both object-compatible and object-incompatible 
actions induce strong activation in the Action Network, including 
premotor-parietal and temporo-occipital areas. This activation pattern is 
even intensified by object-incompatible action (B) where participants have 
to rely on manipulation information to recognize the action. 
Correspondingly, activity is significantly enhanced in fusiform gyrus when 
object information is valid (C). 



hypothesis, which was based on the role of inferior parietal lobe in 
the appraisal of pragmatic implications provided by objects. Left 
pMTG was not hypothesized and will be discussed as a post-hoc 
finding. 

The left IPL activation was restricted to the anterior bank of 
the intraparietal sulcus (alPS) and did not encroach into supra- 
marginal gyrus (SMG). This is an important observation, since 
these two areas have distinct functions, as implicated by research 
in their putative homologues in the macaque, AIP and PF, respec- 
tively (Committeri et al, 2007; McGeoch et al, 2007). The latter 
mediates between PMv and pSTS in a network coined "mirror 



neuron system" or MNS for both action observation and action 
execution (as lucidly outlined in Keysers and Perrett, 2004), 
whereas the former provides PMv with a pragmatic description 
of objects (Fagg and Arbib, 1998). The core difference here is that 
neurons in AIP already respond to objects even when not manip- 
ulated, whereas PF neurons are particularly tuned to the sight 
of the experimenter grasping and manipulating objects (Gallese 
et al, 2002). 

This difference seems particularly relevant in the context of the 
present findings, as the conjunction contrast aimed to tap only 
the parametric effects of object-evoked action knowledge, inde- 
pendent of the object-compatibility of the observed manipulation. 
It makes perfect sense that the parametric action codes contrast 
did not identify SMG (as putative human PF-homolog), because 
activation that was caused by observation was accounted for by 
the main effect action vs. rest (Figure 5), i.e., it was canceled 
out in the parametric action codes contrast. Notably, exploita- 
tion of action codes was reflected by extension of activation 
into SMG that we found only for object-compatible actions, as 
will be discussed later. Thus, our findings perfectly corroborate 
the assumption of a functional dissociation or relative weight- 
ing of AlP/alPS reflecting object-related action information and 
PF/SMG reflecting the observation of object manipulation. 

Human and macaque data converge with regard to the 
manipulation-related role of anterior intraparietal cortex. The 
role of macaque AIP in providing pragmatic object descriptions 
has been related to "hand manipulation neurons" (Gardner et al., 
2007a) in this region and to the encoding of context-specific 
hand grasping movements to perceived objects (Gallese et al., 
1994; Murata et al, 1996; Baumann et al, 2009). Human left 
aIPL is selectively activated during the explicit retrieval of specific 
ways of grasping tools (Chao and Martin, 2000) and manip- 
ulating objects (Kellenbach et al, 2003). Using an interaction 
design implementing two cue types (naming and pantomiming) 
and two response triggers (objects and actions), Rumiati et al. 
(2004) showed that the left aIPL is particularly active for the 
transforming objects into skilled object manipulation. A recent 
fMRI study showed that activity in human alPS reflects the rela- 
tionship between object features and grasp type, as in macaques 
(Begliomini et al., 2007). Also paralleling macaque data, alPS is 
particularly enhanced when object information is to be trans- 
ferred between the visual and the tactile modality (Grefkes et al., 
2002). Our results crucially extend these findings, showing that 
activity in alPS increases with the mere implication of more 
possible actions, i.e., the more visual properties of the objects 
are mentally transferred to different, merely imagined tactile 
properties. 

The present study did not distinguish between seman- 
tic/conceptual ("what") and procedural/motor ("how to") rep- 
resentations triggered by the sight of objects, and its perfectly 
possible that both are automatically evoked. However, there is 
some evidence that aIPL is more related to the "how to" knowl- 
edge related to objects. For instance, Boronat et al. (2005) asked 
participants to determine whether two given objects are manipu- 
lated similarly (e.g., a piano and a laptop keyboard) or serve the 
same function (e.g., a box of matches and a lighter). Only the left 
IPL was more engaged during judgments on manipulation than 
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during judgments on object function (cf. Kellenbach et al, 2003 
for parallel findings). 

Patient studies support this interpretation, showing that dam- 
age to the left IPL can result in an inability to recognize and 
produce precise hand postures associated with familiar objects 
while functional knowledge of objects seems spared (Buxbaum 
and Saffran, 2002; Buxbaum et al., 2003). Binkofski and Buxbaum 
(2013) proposed that two action systems have to be distinguished 
in the dorsal stream: a bilateral dorso-dorsal "grasp" stream link- 
ing superior parietal to dorsal premotor sites for reaching and 
grasping objects based on their size, shape or orientation; and 
a ventro-dorsal "use" system linking inferior parietal to ventral 
premotor sites for skilled functional object use. 

In the present study, objects varied with regard to the num- 
ber of implied actions, and thereby ways to use the objects, but 
of course, also in the way to grasp them. Although our para- 
metric approach — object-evoked action options — tapped a very 
subtle source of variance in our stimulus material (videos), this 
approach did not allow distinguishing between automatically 
evoked representations of object-related ways of manipulating, 
and object-related ways of grasping (i.e., affordances). However, 
participants were required to recognize the observed actions, and 
hence could not solely rely on the observed kind of grasping; 
rather, they had to exactly analyze the way of subsequent usage to 
determine the observed action with confidence. Moreover, find- 
ing PMv and aIPL to increase with the number of active action 
codes points to the ventro-dorsal "use" system rather than to the 
dorso-dorsal "grasp" stream. 

Left pMTG showed up in the action vs. rest contrast, as 
expected, as left pMTG is mostly seen in action observation, and 
also for tool perception (cf. Introduction). However, just as left 
aIPL, pMTG was also found to positively co-vary with the number 
of object-implied actions (Figure 4A). Fusiform gyrus, pMTG, 
and aIPL are considered sensitive to the three types of informa- 
tion required for identification of tools: their visual form, the 
typical motion with which they move when we use them, and 
the way they are manipulated, respectively (Beauchamp et al., 
2002; Mahon et al., 2007). Following this view, we suggest pMTG 
and aIPL both co-varied in activation with the number of active 
action codes, because action codes differed not only with regard 
to the way we use objects, but also in the way the object moves 
while used. For instance, when participants saw the actor han- 
dling a knife and an apple, the automatically evoked action codes 
included two sorts of knife manipulation, but also the corre- 
sponding two sorts of knife motion. Of course, the visual form of 
the knife was invariant, fitting to the fact that action codes showed 
no effect in fusiform gyrus. Note that other authors have put for- 
ward that pMTG rather than being a motion-coding area, repre- 
sents conceptual object knowledge (Johnson-Frey, 2004; Fairhall 
and Caramazza, 2013). There might be also subtle regional dif- 
ferences in functions, as the posterior temporal region contains a 
variety of functionally specialized areas. 

In fact, pMTG refers to an only vague macroanatomical def- 
inition of a cortical region that lies in direct vicinity of func- 
tional related areas. The peak coordinates of the left MTG in 
our study were at Talairach x = —47, y = —67, z = 9, which is 
nearly identical to peak coordinates of the extrastriate body area 



(EBA; x = ±47.2, y = —66.7, z = 4.7) when averaging across 13 
recent fMRI studies (Downing et al, 2006a,b; Taylor et al, 2007; 
Myers and Sowden, 2008). Moreover, human motion selective 
area hMT (Greenlee, 2000; Peuskens et al., 2005) overlaps with 
EBA (e.g., Downing et al., 2007; Taylor et al, 2007). Although 
the parametric increase of activity in EBA or hMT in the cur- 
rent experimental design cannot be due to demands on body 
part and motion perception, it could reflect the range of move- 
ments and body postures associated with a given object. On the 
one hand, EBAs contribution in the processing of body posture 
(Downing et al, 2006b) could be required here as referring to 
typical hand postures and configurations indicative of the manip- 
ulations applicable to an object. On the other hand, hMT is 
engaged in the processing of complex motion patterns (Peuskens 
et al, 2005), but also in motion as merely implied or announced 
by hand postures or objects (Kourtzi and Kanwisher, 2000). 
Interestingly, both hMT and EBA, together with pSTS sensitive 
to the perception of biological motion, were found to adapt to 
the repetition of observed actions even when novel exemplars of 
object manipulation were shown, suggesting a role of these areas 
in the representation of the type of manipulation rather than its 
particular instantiation (Kable and Chatterjee, 2006). Our find- 
ings fit to this notion, as the parametric effect of action codes in 
pMTG and aIPL was independent of the actual observation of one 
of these actions (revealed by the conjunction of both). 

EXPLOITING OBJECT-RELATED KNOWLEDGE TO RECOGNIZE ACTIONS 

Only object-compatible, not object-incompatible actions 
matched one of the action codes supposedly evoked by the 
sight of the involved objects. Thus, the NAC quantified the 
constraint imposed onto recognizing object-compatible, not 
object-incompatible actions: the actually observed action was 
one out of about one, two or three expectable actions. As 
hypothesized, PMv activity positively co-varied with the NAC 
in object-compatible but not incompatible actions. We found 
this activation in both hemispheres, together with corresponding 
activation foci in anterior IPL, bilateral lateral occipital cortex 
extending into pMTG in the right hemisphere, and bilateral mid- 
insula (Figure 4B). Activity in pIPS was bilateral, but pronounced 
in the right hemisphere spanning from a ventral postcentral 
region and anterior SMG up to the horizontal segment of the 
IPS. The fact that left aIPL, the area that we had found for the 
parametric effect of the NAC, surfaced in this interaction contrast 
as well indicate that it was dominant, though not specific, for 
object-compatible actions. 

As outlined in the Introduction, we take this network to 
reflect the fronto-parietal reinforcement of object-implied action 
options while tracking the unfolding action. Here, the observed 
action matched one of actions the observer was expecting due to 
the observed object or set of objects. In this case, and only then, 
PMv reinforced the matching action manipulation in IPL, and 
the matching tool motion in pMTG. Further extrastriate visual 
activation located in the right cuneus may point to modulations 
going even further downstream. 

Importantly, this interaction contrast tapped only into areas 
whose activation increased with the competition load between 
object-evoked action options. This effect was observed not only 
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at the frontal component of the Action Network, but also at the 
corresponding parietal and posterior temporal sites. Thus, rein- 
forcement increases activation also at the targets in the posterior 
brain. It is well-known that frontal and parietal/temporal areas 
interact for selective purposes in attention, with the latter provid- 
ing "bottom-up" externally driven, perceptual input on which the 
frontal areas exert a "top-down" selective modulation for goal- 
directed cognition and behavior (Frith, 2001; Bar, 2003; Pessoa 
et al., 2003). For sure, the relevant parietal and temporal areas 
themselves provide highly integrated information of the stimulus. 
That is, they build rather a "mid" level between frontal and lower 
visual areas, exerting "top-down" biasing signals on the latter as 
well (Kastner and Ungerleider, 2001). 

Premotor-parietal-temporal activation patterns during action 
observation have been suggested to reflect a re-activation of 
actions stored in memory (Decety and Ingvar, 1990; Jeannerod, 
1999). Our findings specify this formula by showing that dur- 
ing action observation, the premotor, parietal, and temporal 
components of this network differ with regard to their sensitiv- 
ity of object-implied actions: Unlike IPL and pMTG, PMv was 
only sensitive for the exploitation of competing implied actions, 
but not for the mere number of implied actions. While IPL 
and pMTG reflected the action options both as evoked by the 
sight of the objects (bottom-up) and as competition resolved 
by frontal biasing signals (top-down), PMv was indifferent with 
regard to the former: it showed for the number of action codes 
effect for the interaction between, but not for the conjunction of, 
object-compatible and object-incompatible actions. 

In order to understand and interpret this finding, it helps 
to consider three converging results in the present study: 
PMv was present in both object-compatible and object- 
incompatible action (conjunction contrast, Figure 5), more pro- 
nounced in object-incompatible as in object-compatible actions 
(masked direct contrast, Figure 5) (H3), and driven by object- 
evoked action options in object-compatible but not in object- 
incompatible actions (interaction of action codes and object 
compatibility, Figure 4B). This data pattern suggests that PMv 
not only registers object-evoked action representations, as aIPL 
and pMTG do, but also dynamically applies these internal action 
representations, either in order to adapt to, or to predict, the 
ongoing action (cf. Schubotz, 2007). 

As to the parietal activation revealed in the interaction con- 
trast, it was found to extend from anterior to posterior IPS in 
the right hemisphere. Why was pIPS activity so pronounced for 
the right hemisphere? Mruczek et al. (2013) recently reported 
that tools evoke stronger responses than non-tools in an anterior 
intraparietal region. Authors suggest that posterior IPS encode 
features common to any graspable object (including tools), 
whereas anterior IPS integrate this grip-relevant information with 
"experience-dependent knowledge of action associations, affor- 
dances, and goals, which are uniquely linked to tools" (Mruczek 
et al., 2013, p. 2892). Coordinates of activation maxima in poste- 
rior IPS were most closely located to those related to macaque area 
MIP (Grefkes and Fink, 2005). MIP is suggested to be involved 
in coordination of hand movements and visual targets (Eskandar 
and Assad, 2002), particularly in transforming the spatial coor- 
dinates of a target into a representation that is exploited by the 
motor system for computing the appropriate movement vector 



(Cohen and Andersen, 2002). Interestingly, these computations 
take place even in advance of the motion execution itself (Johnson 
et al., 1996) and hence point to a role of MIP in the detection 
of movement errors and their correction already on the basis of 
internal models (Kalaska et al., 2003). More recent studies spec- 
ify this region as providing tactile information to circuits linking 
anterior intraparietal to ventral premotor regions, giving on-line 
feedback needed for goal-directed hand movements (Gardner 
et al, 2007a,b). 

This computational profile was perfectly reflected in the 
increase of activity in this region reported here, when pragmatic 
object-implied constraints on expectable manipulations could be 
integrated with the currently unfolding action. Also the notion 
of detection of movement errors and their correction on the 
basis of internal models (Kalaska et al, 2003) fits very well to 
the present finding, as our parametric contrast pinpointed the 
competition load between action options. Thus, when multiple 
action options were implied by the perceived object or object set, 
and hence represented as multiple internal models of potentially 
observable manipulations, pIPS may contribute to the detection 
of discrepancies between expected and observed manipulations. 

To finally address activation in the mid-insula, this region 
relays tactile information from the somatosensory cortex to the 
frontal cortex (Burton and Sinclair, 2000). Activity was located at 
the posterior short insula gyrus, which is delimited by the precen- 
tral and central insular sulci. This dysgranular region has connec- 
tions to SI and SII (cf. Guenot et al., 2004 for review). Together 
with SII and SMG, the mid-insula is suggested to play a crucial 
role in tactile object recognition, and to integrate somatosensory 
information to provide a coherent image of an object appropri- 
ate for cognitive action (Reed et al., 2004; Milner et al, 2007). 
Since we found this region to positively co-vary in activity with 
the number of object-implied expectable manipulations, but only 
when the observed action matched one of them, we speculate that 
the enhancement of the matching action comprised also a tactile 
representation of the observed object manipulation. 

OBSERVING OBJECT-INCOMPATIBLE ACTIONS: OBJECTS EVOKE 
ACTION OPTIONS THAT DO NOT FOSTER ACTION RECOGNITION 

In the present study, videos showing object-incompatible actions 
(i.e., pantomimes with incompatible objects) were employed as 
a control condition that served to tell apart effects that could be 
only due to the sight of objects (common to object-compatible 
and incompatible actions) from effects that could be only due 
their manipulation (different between object-compatible and 
incompatible actions). Objects and object sets were always rem- 
iniscent of valid action options, both in object-compatible and 
object-incompatible actions. Moreover, object-incompatible and 
object-compatible actions were presented randomly intermixed 
and each occurred with equal probability of 0.5. Together, these 
design features provoked, as intended, an initial analysis of object 
information and an attempt to match the observed actions on one 
of the automatically evoked action code. 

To be sure, object-incompatible action is certainly more than 
just some kind of "incomplete" action, and there are posi- 
tive effects of object-incompatible action, i.e., activations that 
come in addition to what we see during observation of object- 
compatible action. We have investigated these effects elsewhere 
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(Schubotz and von Cramon, 2009). Replicating these findings, 
object-compatible and object-incompatible action observation 
yielded highly similar activation patterns in the resting contrast, 
with significant differences only in the emphasis of different parts 
of the Action Network (cf. conjunction in Figure 5). Thus, dif- 
ferences in our parametric analyses in object-compatible and 
object-incompatible action could not be due to principally absent 
activations in the object-incompatible condition, i.e., "positive 
effects" in the object-compatible action condition cannot be just 
due to "negative effects" in object-incompatible actions. 

It is important to note that humans are perfectly able to decode 
actions from object-incompatible manipulations, even from early 
childhood on (Fein, 1981). Also in the present study, participants 
performed as well in object-incompatible trials (4.8% errors, 
1232 ± 69 ms reaction time) as in object-compatible action trials 
(5.8% errors, 1192 ± 61ms reaction time). Saying that partic- 
ipants failed to exploit object-evoked action representations in 
the case of object-incompatible actions thus does not mean that 
they failed in the task, but rather, that their strategy had to be 
adapted according to the stimulus. Object-incompatibility was 
revealed by a mismatch between the currently active action codes 
and the actually observed manipulation. As object information 
was invalid here, observers had to entirely rely on the analysis 
of hand movements when trying to decode the currently pur- 
sued action. In accordance with this suggestion, and replicating 
a previous study (Schubotz and von Cramon, 2009), we found 
that the entire Action Network engaged in manipulation recog- 
nition (PMv, alPS, pMTG) enhanced for object-incompatible as 
compared to object-compatible actions (cf. Hypothesis H3). 

However, we also reasoned that action codes should not result 
in any significant effect on object-incompatible actions since they 
cannot help to constrain the recognition process. However, the 
novel and striking finding here was that activity in an area com- 
prising left pSTS and TPJ did increase with the NAC. More 
specifically, this activation was located in the horizontal pos- 
terior segment of the pSTS, extending toward the ascending 
posterior segment, and hence comprised a temporal-parietal- 
occipital junction (BA 37, 39 and 19) (Figure 4D and blue spot in 
Figure 4B). Activation was left-lateralized, corresponding to the 
processing of information from the right visual field, and, in the 
present study, the dominant (right) hand of the actor. This fits 
well with the experimental setting as, to an observer, motion and 
posture of the dominant hand is more informative than that of 
the non-dominant hand; the latter typically holds and stabilizes 
the object while the former performs the relevant manipulations. 
Focus on the right visual field was also indicated by increased 
activation in left cuneus (cf. Machner et al., 2009). 

These effects of an increasing NAC were only found for object- 
incompatible actions and indicated that, although object infor- 
mation was in fact not usable here, the expectations of particular 
hand movements announced by the objects still affected fur- 
ther stimulus analysis. Note that the more action options were 
evoked by the object, the lower were the constraints on the 
to-be-expected manipulations. At the same time, the probabil- 
ity increased that the actually observed manipulation eventually 
matches one of the pre-activated actions: When you expect one 
out of three potential actions to occur, a fourth and unexpected 



action is more difficult to detect than in a case when you expect 
exactly one specific action to occur. 

With this in mind, we take left pSTS activation to reflect 
the intensified focus on the hand's movements in an attempt to 
decode the displayed action. Interestingly, activation extended 
posteriorly and dorsally into TPJ. Due to its particular func- 
tional profile in attentional orienting as well as in mentalizing 
paradigms (Decety and Lamm, 2007; Mitchell, 2008), TPJ has 
been discussed to have a "where-to" functionality in analogy 
with the spatial "where" functionality of the dorsal stream (Van 
Overwalle, 2009). That is, it responds to externally generated 
behaviors with the aim of identifying the possible end-state of 
these behaviors (cf. discussion in Van Overwalle and Baetens, 
2009). Note that the "end-state" of behavior can be read as being 
actually related to the physical body, as TPJ is related to the 
sensation of the position and the movement of one's own body 
(Blanke et al., 2004). Our findings corroborate this interpreta- 
tion as they showed TPJ activation to proportionally increase 
with the number of expectable end-states of the unfolding action. 
Importantly, TPJ activation was not specific to or indicative of 
object-incompatible (pantomime) perception in general, as the 
TPJ effect was only found for the action code parameter in object- 
incompatible actions, whereas it was absent in the direct contrast 
object-incompatible vs. object-compatible action. 

CONCLUDING REMARKS 

The faculty of understanding what other persons are doing is 
based, among other factors, on the analysis of object and manip- 
ulation information. The present study shows that the action- 
observing brain accurately extrapolates the expectable actions 
from the objects that the actor is handling, and, when detecting a 
match between these expectable actions and the actually observed 
one, subsequently reinforces the matching action against the 
competition of the remaining but unobserved actions. These find- 
ings impressively reflect that object-evoked actions constrain the 
recognition process in action observation 
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